linear neural network
Continuous vs. Discrete Optimization of Deep Neural Networks
Existing analyses of optimization in deep learning are either continuous, focusing on (variants of) gradient flow, or discrete, directly treating (variants of) gradient descent. Gradient flow is amenable to theoretical analysis, but is stylized and disregards computational efficiency. The extent to which it represents gradient descent is an open question in the theory of deep learning.
Expectation Error Bounds for Transfer Learning in Linear Regression and Linear Neural Networks
Liu, Meitong, Jung, Christopher, Li, Rui, Feng, Xue, Zhao, Han
In transfer learning, the learner leverages auxiliary data to improve generalization on a main task. However, the precise theoretical understanding of when and how auxiliary data help remains incomplete. We provide new insights on this issue in two canonical linear settings: ordinary least squares regression and under-parameterized linear neural networks. For linear regression, we derive exact closed-form expressions for the expected generalization error with bias-variance decomposition, yielding necessary and sufficient conditions for auxiliary tasks to improve generalization on the main task. We also derive globally optimal task weights as outputs of solvable optimization programs, with consistency guarantees for empirical estimates. For linear neural networks with shared representations of width $q \leq K$, where $K$ is the number of auxiliary tasks, we derive a non-asymptotic expectation bound on the generalization error, yielding the first non-vacuous sufficient condition for beneficial auxiliary learning in this setting, as well as principled directions for task weight curation. We achieve this by proving a new column-wise low-rank perturbation bound for random matrices, which improves upon existing bounds by preserving fine-grained column structures. Our results are verified on synthetic data simulated with controlled parameters.
On the spectral bias of two-layer linear networks
The most forceful driver of advancements in the field of Machine Learning over the past decades has been the success of deep neural networks.
Global Convergence of Gradient Descent for Deep Linear Residual Networks
It is motivated by avoiding stable manifolds of saddle points.
Provable Acceleration of Nesterov's Accelerated Gradient for Rectangular Matrix Factorization and Linear Neural Networks
We study the convergence rate of first-order methods for rectangular matrix factorization, which is a canonical nonconvex optimization problem.
